Data Ingestion using Amazon Kinesis Data Streams

Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store data streams at any scale. The types of data that you can stream using Amazon Kinesis Data Stream includes IT infrastructure log data, application logs, social media, market data feeds, to name a few. Data Streams enable transfer of data from multiple sources at a steady but high speed to provide real-time insights by the processing and analysis of big data in motion. A typical use of streaming data can be seen in a ride-haling application where you have to combine data about user, location, destination, and traffic conditions. Matching riders with available drivers in terms of proximity, pricing and wait times can only be achieved by processing streaming data and providing the required real-time insights.

In Data Pipeline Studio it is possible to create a data ingestion pipeline using Amazon Kinesis Data Streams. You can use the data integration tool Databricks to ingest the data either into an Amazon S3 data lake or into a Snowflake data lake.

Data Source	Data Integration	Data Lake	URL
Amazon Kinesis Data Streams	Databricks	Amazon S3	Data Ingestion using Amazon Kinesis Data Streams with S3 Data Lake
Amazon Kinesis Data Streams	Databricks	Snowflake	Data Ingestion using Amazon Kinesis Data Streams with Snowflake Data Lake

What's next? Data Ingestion using Amazon Kinesis Data Streams with S3 Data Lake